Conversation
|
@bugout-dev check
|
zomglings
left a comment
zomglings
left a comment
There was a problem hiding this comment.
Reconsidered crawler architecture.
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def parser_queries_execute_handler(args: argparse.Namespace) -> None: |
There was a problem hiding this comment.
After discussing with @kompotkot and @Andrei-Dolgolev:
We realized that we want this crawler to go through the Query API like any other user would.
The plan is to remove this mooncrawl/queries_crawler code and update the Moonstream Python client to provide this functionality similarly to how we used it in autocorns biologist:
https://github.com/bugout-dev/autocorns/blob/cf00fb492de254821730a256d238d5a332810db6/autocorns/biologist.py#L371
We can remove the existing Moonstream Python client, bump the client version, and publish the new client.
|
We need to cherry pick from this PR later. |
|
Although we will not use The We will revisit this after our current batch of urgent work. |
2 participants
Changes
Fetch query from journal, validate query, execute query and push to bucket if required.
How to test these changes?
Tested locally
Related issues